Skip to content

feat(media): add sharded media object-key paths - #4533

Open
bradseiler wants to merge 17 commits into
mainfrom
seiler/media-layout-migration
Open

feat(media): add sharded media object-key paths#4533
bradseiler wants to merge 17 commits into
mainfrom
seiler/media-layout-migration

Conversation

@bradseiler

@bradseiler bradseiler commented Aug 3, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Why

Media payload keys are currently flat, so high-volume traffic lacks hash-leading S3 prefix entropy. Existing deployments also need a safe, explicit rollout that preserves old reads and rollback behavior without changing storage behavior merely by upgrading.

What

  • moves media payloads to media/<sha[0:2]>/<sha[2:4]>/<community>/<filename> while leaving Git/CAS namespaces unchanged
  • adds BUZZ_MEDIA_MIGRATION_PHASE with three complete media object-key path policies:
    • legacy-only (default): read and write only legacy paths, preserving existing deployment behavior
    • dual-read-and-write: prefer sharded reads with legacy fallback and write both paths
    • sharded-only: read and write only sharded paths
  • defines the existing-deployment rollout as legacy-only → dual-read-and-write → backfill → sharded-only
  • recommends that new deployments start with sharded-only before their first upload, avoiding any future backfill or legacy cleanup
  • adds sharded-first/legacy-fallback read telemetry, strict bucket classification, logical billing deduplication, duplicate-layout gauges, and authoritative payload keys in upload records
  • includes buzz-media-layout-backfill and buzz-media-layout-delete-legacy in both relay image targets, with bounded paging, checkpoints, request-rate limiting, idempotency, destination verification, dry-run, and destructive confirmation
  • documents new-install and existing-deployment flows and provides Kubernetes Job examples

Validation

  • cargo test -p buzz-media --all-targets: 119 passed; live-MinIO test ignored
  • cargo test -p buzz-relay --lib config::tests::: 32 passed
  • cargo test -p buzz-relay --lib storage_sweep: 15 passed
  • cargo clippy -p buzz-media -p buzz-relay --all-targets -- -D warnings: clean
  • cargo fmt --all -- --check, desktop Tauri fmt, and git diff --check: clean
  • pre-push hooks after merging current origin/main: Rust, desktop, Tauri, mobile, and typecheck suites passed
  • Helm values/test YAML and values schema JSON parse cleanly; Helm CLI is unavailable locally

Generated with Goose

npub128x7j3pwgm4vs8yra3c42fcgcwcvh94g3luwzkqa376du2q6l0esqcrwch added 6 commits August 3, 2026 10:13
npub128x7j3pwgm4vs8yra3c42fcgcwcvh94g3luwzkqa376du2q6l0esqcrwch
feat(media): add sharded compatibility reads
a021939 
Co-authored-by: npub128x7j3pwgm4vs8yra3c42fcgcwcvh94g3luwzkqa376du2q6l0esqcrwch <51cde9442e46eac81c83ec71552708c3b0cb96a88ff8e1581d8fb4de281afbf3@buzz.block.builderlab.xyz>
Signed-off-by: npub128x7j3pwgm4vs8yra3c42fcgcwcvh94g3luwzkqa376du2q6l0esqcrwch <51cde9442e46eac81c83ec71552708c3b0cb96a88ff8e1581d8fb4de281afbf3@buzz.block.builderlab.xyz>
npub128x7j3pwgm4vs8yra3c42fcgcwcvh94g3luwzkqa376du2q6l0esqcrwch
feat(media): observe sharded migration reads
786f4c9 
Classify sharded payload keys in storage sweeps, preserve physical totals, and deduplicate logical legacy/sharded copies. Export read resolution, fallback, and duplicate-layout metrics.

Co-authored-by: npub128x7j3pwgm4vs8yra3c42fcgcwcvh94g3luwzkqa376du2q6l0esqcrwch <51cde9442e46eac81c83ec71552708c3b0cb96a88ff8e1581d8fb4de281afbf3@buzz.block.builderlab.xyz>
Signed-off-by: npub128x7j3pwgm4vs8yra3c42fcgcwcvh94g3luwzkqa376du2q6l0esqcrwch <51cde9442e46eac81c83ec71552708c3b0cb96a88ff8e1581d8fb4de281afbf3@buzz.block.builderlab.xyz>
npub128x7j3pwgm4vs8yra3c42fcgcwcvh94g3luwzkqa376du2q6l0esqcrwch
feat(media): include payload keys in upload records
49bf37d 
Co-authored-by: npub128x7j3pwgm4vs8yra3c42fcgcwcvh94g3luwzkqa376du2q6l0esqcrwch <51cde9442e46eac81c83ec71552708c3b0cb96a88ff8e1581d8fb4de281afbf3@buzz.block.builderlab.xyz>
Signed-off-by: npub128x7j3pwgm4vs8yra3c42fcgcwcvh94g3luwzkqa376du2q6l0esqcrwch <51cde9442e46eac81c83ec71552708c3b0cb96a88ff8e1581d8fb4de281afbf3@buzz.block.builderlab.xyz>
npub128x7j3pwgm4vs8yra3c42fcgcwcvh94g3luwzkqa376du2q6l0esqcrwch
feat(media): add configurable sharded writes
9d8a7ac 
Co-authored-by: npub128x7j3pwgm4vs8yra3c42fcgcwcvh94g3luwzkqa376du2q6l0esqcrwch <51cde9442e46eac81c83ec71552708c3b0cb96a88ff8e1581d8fb4de281afbf3@buzz.block.builderlab.xyz>
Signed-off-by: npub128x7j3pwgm4vs8yra3c42fcgcwcvh94g3luwzkqa376du2q6l0esqcrwch <51cde9442e46eac81c83ec71552708c3b0cb96a88ff8e1581d8fb4de281afbf3@buzz.block.builderlab.xyz>
npub128x7j3pwgm4vs8yra3c42fcgcwcvh94g3luwzkqa376du2q6l0esqcrwch
docs(media): document staged key migration
ff67621 
Keep self-hosted upgrades on legacy writes by default, expose the write-layout gate through Helm and Compose, and document the explicit legacy-to-dual-to-sharded migration sequence.

Co-authored-by: npub128x7j3pwgm4vs8yra3c42fcgcwcvh94g3luwzkqa376du2q6l0esqcrwch <51cde9442e46eac81c83ec71552708c3b0cb96a88ff8e1581d8fb4de281afbf3@buzz.block.builderlab.xyz>
Signed-off-by: npub128x7j3pwgm4vs8yra3c42fcgcwcvh94g3luwzkqa376du2q6l0esqcrwch <51cde9442e46eac81c83ec71552708c3b0cb96a88ff8e1581d8fb4de281afbf3@buzz.block.builderlab.xyz>
npub128x7j3pwgm4vs8yra3c42fcgcwcvh94g3luwzkqa376du2q6l0esqcrwch
feat(media): add phased layout migration
6089cf0 
Consolidate read and write policy behind one upgrade-safe phase and ship guarded maintenance binaries for backfill and legacy cleanup.

Co-authored-by: npub128x7j3pwgm4vs8yra3c42fcgcwcvh94g3luwzkqa376du2q6l0esqcrwch <51cde9442e46eac81c83ec71552708c3b0cb96a88ff8e1581d8fb4de281afbf3@buzz.block.builderlab.xyz>
Signed-off-by: npub128x7j3pwgm4vs8yra3c42fcgcwcvh94g3luwzkqa376du2q6l0esqcrwch <51cde9442e46eac81c83ec71552708c3b0cb96a88ff8e1581d8fb4de281afbf3@buzz.block.builderlab.xyz>
This was referenced Aug 3, 2026
Closed
Closed
Closed
npub128x7j3pwgm4vs8yra3c42fcgcwcvh94g3luwzkqa376du2q6l0esqcrwch and others added 4 commits August 3, 2026 10:27
npub128x7j3pwgm4vs8yra3c42fcgcwcvh94g3luwzkqa376du2q6l0esqcrwch
chore(desktop): update media dependency lock
4be3093 
Co-authored-by: npub128x7j3pwgm4vs8yra3c42fcgcwcvh94g3luwzkqa376du2q6l0esqcrwch <51cde9442e46eac81c83ec71552708c3b0cb96a88ff8e1581d8fb4de281afbf3@buzz.block.builderlab.xyz>
Signed-off-by: npub128x7j3pwgm4vs8yra3c42fcgcwcvh94g3luwzkqa376du2q6l0esqcrwch <51cde9442e46eac81c83ec71552708c3b0cb96a88ff8e1581d8fb4de281afbf3@buzz.block.builderlab.xyz>
npub128x7j3pwgm4vs8yra3c42fcgcwcvh94g3luwzkqa376du2q6l0esqcrwch
fix(media): use explicit media key prefix
7bfd885 
Co-authored-by: npub128x7j3pwgm4vs8yra3c42fcgcwcvh94g3luwzkqa376du2q6l0esqcrwch <51cde9442e46eac81c83ec71552708c3b0cb96a88ff8e1581d8fb4de281afbf3@buzz.block.builderlab.xyz>
Signed-off-by: npub128x7j3pwgm4vs8yra3c42fcgcwcvh94g3luwzkqa376du2q6l0esqcrwch <51cde9442e46eac81c83ec71552708c3b0cb96a88ff8e1581d8fb4de281afbf3@buzz.block.builderlab.xyz>
@bradseiler
fix(media): preserve legacy layout by default
5f9a4e2 
Signed-off-by: Brad Seiler <seiler@squareup.com>
@bradseiler
Merge remote-tracking branch 'origin/main' into seiler/media-layout-m…
166ffe3 
…igration

* origin/main: (95 commits)
  fix(desktop): virtualize channel member lists (#4991)
  fix(desktop): enforce owner-only access in internal builds (#4053)
  test(desktop): match attachment button label (#4993)
  fix(acp): pace observer telemetry at 1/s with per-channel batch envelopes (#4917)
  fix(desktop): enable the content security policy (#4614)
  fix(mobile): merge relay recounts with locally seen thread replies (#4633)
  fix(desktop): enable message editing in Inbox (#2198)
  relay: fuzz WebSocket 1012 restart-close timing on graceful drain (BUZZ_DRAIN_JITTER_MS) (#4542)
  fix(desktop): outline the selected community (#4969)
  fix(desktop): clamp thread panel to channel surface (#4965)
  style(messages): increase username contrast (#4948)
  fix(desktop): rename generic attachment action from 'Attach image' to 'Attach file' (#2381) (#4304)
  fix(reactions): support max-length custom emoji (#3833)
  feat(desktop): allow leaving your final community (#3621)
  fix(buzz-agent): recover from context-window 400s instead of sticking (#4946)
  docs(persona-pack): fix stale desktop import instructions (#4500)
  fix(desktop): route macos notification clicks (#4799)
  feat(mobile): sync themes per community (#3767)
  feat(desktop): sync themes per community (#3653)
  feat(desktop): cap OpenClaw agent parallelism at 5 (#4019)
  ...

Signed-off-by: Brad Seiler <seiler@squareup.com>
@bradseiler bradseiler changed the title feat(media): add phased S3 layout migration feat(media): add sharded media object-key paths Aug 6, 2026
@bradseiler
fix(media): simplify object-key path rollout
e2882f6 
Signed-off-by: Brad Seiler <seiler@squareup.com>
@bradseiler
Merge remote-tracking branch 'origin/main' into seiler/media-layout-m…
e0503d5 
…igration

* origin/main:
  fix(bench): mention the orchestrator by pubkey when posting the task (#5136)
  feat(relay): accept kind:30179 private managed-agent events at ingest (#5133)
  fix(media): require authenticated reads (#4610)
  fix(desktop): preserve authoritative agent avatars (#4984)
  fix(desktop): next/back navigation during key creation onboarding (#4978)
  Alert community owners and admins when a new key joins (#4900)
  fix(desktop): prevent sidebar prefs from reverting on stale-localStorage boot (#5086)
  chore(hooks): run desktop typecheck in pre-push (#5110)
  feat(identity): recover desktop identity from a signed-in phone (#4845)
  fix(buzz-agent): classify read timeouts distinctly in LLM error messages (#4959)
  Refine agent runtime controls (#5026)
  test(desktop): await thread scroll anchor (#3174)
  Improve desktop mobile pairing flow (#5024)
  feat(desktop): show selected community in rail (#5000)
  fix(desktop): stop rate-limited reconnect backfill from tearing down the authenticated socket (#4990)
  fix(desktop): skip native notifications outside app bundles (#5004)
  ci: prove the relay-driven mesh lifecycle — discover, join, infer, deny — with real nodes (#3862)

Signed-off-by: Brad Seiler <seiler@squareup.com>

# Conflicts:
#	deploy/charts/buzz/templates/deployment.yaml
#	deploy/charts/buzz/tests/render_test.yaml
#	deploy/charts/buzz/values.schema.json
#	deploy/charts/buzz/values.yaml
@bradseiler
fix(media): make upload short-circuit blob key structural
731dd94 
Replace the Option + expect("checked above") pattern in both upload
short-circuit paths with a filtered if-let binding, so the presence of
the blob key is guaranteed structurally instead of by a comment on a
non-local invariant. Also collapse the longhand match on
existing_write_key into `?`. No behavior change.

Signed-off-by: Brad Seiler <seiler@squareup.com>
@bradseiler
docs(media): warn on phase rollback and start compose sharded
80ee9a9 
Document that returning to legacy-only after accepting uploads in
sharded-only makes sharded-layout objects unreadable until the phase is
raised again. Ship sharded-only in the Compose example environment so
fresh stacks never need a backfill, and correct the chart README to
reflect that only the chart default remains legacy-only for upgrade
safety.

Signed-off-by: Brad Seiler <seiler@squareup.com>
@bradseiler
bradseiler marked this pull request as ready for review August 7, 2026 04:55
@bradseiler
bradseiler requested a review from a team as a code owner August 7, 2026 04:55
coder @bradseiler
fix(media): verify backfilled destination bytes
828b51e 
Co-authored-by: Brad Seiler <seiler@squareup.com>
Signed-off-by: Brad Seiler <seiler@squareup.com>
@bradseiler
bradseiler force-pushed the seiler/media-layout-migration branch from ada0dd3 to 828b51e Compare August 7, 2026 05:13
coder and others added 2 commits August 7, 2026 01:16
coder @bradseiler
fix(media): verify legacy cleanup bytes
382b3ce 
Co-authored-by: Brad Seiler <seiler@squareup.com>
Signed-off-by: Brad Seiler <seiler@squareup.com>
coder @bradseiler
fix(media): read payloads through migration candidates
963c626 
Co-authored-by: Brad Seiler <seiler@squareup.com>
Signed-off-by: Brad Seiler <seiler@squareup.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@bradseiler